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Abstract 


In psychology and neuroscience, multiple object tracking (MOT) refers to the ability of humans and other animals 
to simultaneously monitor multiple objects as they move. It is also the term for a laboratory technique used to 
study this ability. In an MOT study, a number of identical moving objects are presented on a display. Some of the 
objects are designated as targets while the rest serve as distractors. Study participants try to monitor the changing 
positions of the targets as they and the distractors move about. At the end of the trial, participants typically are 
asked to indicate the final positions of the targets. The results of MOT experiments have revealed dramatic limi- 
tations on humans' ability to simultaneously monitor multiple moving objects. For example, awareness of features 
such as colour and shape is disrupted by the objects' movement. 


Keywords: visual perception 


Introduction 


History 


In the 1970s, the researcher Zenon Pylyshyn postu- 
lated the existence of a "primitive visual process" in the 
human brain capable of “indexing and tracking fea- 
tures or feature clusters”. Using this process, cognitive 
processes can continuously refer to, or "track", objects 
despite the movement of the objects causing them to 
stimulate different visual neurons over time." Data 
collected with Pylyshyn's multiple object tracking 
(MOT) protocols and published in 1988 provided the 
first formal demonstration that the mind can keep 
track of the changing positions of multiple moving ob- 
jects. 


Asa specific theory of this ability, Pylyshyn proposed 
FINST (Fingers of Instantiation) theory, which claims 
that tracking is mediated by a fixed set of discrete 
pointers. While FINST theory has been very influential, 
many studies have found evidence that seems incon- 
sistent with the theory.” 
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Procedure 


Figure 1 | Sequence of events in a typical MOT task. 
Teeeea, CC-BY-SA 4.0. 


A typical MOT study involves the presentation of be- 
tween eight and twelve objects. The participant is told 
to monitor the positions of a subset of the objects, 
which are referred to as targets. Often the targets are 
indicated by being presented initially in a distinct color. 
The targets then become identical in appearance to the 
other, distractor objects. The targets and distractors 
move about the screen for several seconds in an unpre- 
dictable fashion. The participant is then asked to indi- 
cate which of the objects are the targets. The accuracy 
of the participant's judgments indicates whether the 
participant mentally updated the positions of the tar- 
gets as they moved. 


To ensure that the task requires participants to men- 
tally update the targets’ positions, displays are typically 
designed such that object paths cause the targets to 
swap positions with distractors, at least occasionally. 
With that constraint, MOT task variations have been 
designed to probe specific aspects of how the mind 
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tracks moving objects. For example, to compare perfor- 
mance in the left to performance in the right visual 
fields, studies confine some or all the moving objects to 
one of the visual fields.“! To avoid any contribution 
from spatial interference among mental object repre- 
sentations, some studies maintain a minimum distance 
between objects.) Other studies have combined MOT 
with a concurrent task to investigate whether the two 
tasks draw on the same mental resource, and have 
changed target features such as color to assess whether 
study participants update their representations of 
those features. 


Capacity limits 


MOT study results indicate that the number of targets 
that people can track is very limited. This reflects a 
bottleneck in the brain's processing architecture. While 
at the early, sensory stages of visual processing, doz- 
ens of objects may be fully processed, later processes 
such as those associated with cognition have much 
more limited capacity to process visual objects."*! 


The specific number of visual objects that people can 
accurately track varies widely with display parameters, 
contrary to a common belief that people can track no 
more than four or five objects. Even for a fixed set of 
display parameters, rather than there being a clear 
limit, performance falls gradually with the number of 
targets.'°! Such findings undermine Pylyshyn's FINST 
theory that tracking is mediated by a fixed set of dis- 
crete pointers." 


The above limitations appear to stem from processes 
specific to the two cerebral hemispheres. The inde- 
pendence of the limits in the two hemifields is demon- 
strated by findings that when one is tracking the maxi- 
mum number that can be tracked in the left hemifield 
(which is processed by the right cerebral hemisphere), 
one can add targets to the right hemifield (which is 
processed by the left cerebral hemisphere) at little to 
no cost to performance.'!|°! For features other than 
position, capacity seems to be more limited - see the 
"Updating of features other than position" section be- 
low. 


While the tracking capacity limit is largely set sepa- 
rately by the two cerebral hemispheres, a more unified 
and cognitive resource also can contribute to tracking. 
For example, if there is only one target, one can bring 
one's full cognitive abilities to bear, such as in predict- 
ing future positions, to facilitate tracking. When more 
targets are present, these resources may still play a 
role.40] 
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Spatiotemporal limits 


If the objects of a display are not sufficiently widely 
spaced, the objects are difficult to identify and select 
with attention due to spatial crowding,"! which can 
prevent tracking."!""7! High object speeds have a simi- 
lar effect - faster objects are harder to track, and hu- 
mans are completely unable to track objects that move 
sufficiently fast. This "speed limit", however, is much 
slower than the maximum object speed at which hu- 
mans can judge the object's movement direc- 
tion.”*! This dissociation between motion perception 
and object tracking is thought to reflect that direction 
judgments can be based on low-level and local motion 
detector responses that don't register the positions of 
objects. 


As an object's speed is increased, temporal crowding 
can result and prevent tracking well before the track- 
ing speed limit is reached.'*!"“! Temporal crowding re- 
fers to an impairment caused by distractors visiting a 
target's former location within a short interval. The 
phenomenon was revealed in a study with a display 
where distractors were evenly spaced along a circular 
trajectory that was also shared by a target. Partici- 
pants could not track three targets if the locations 
traversed were visited by objects more than three 
times per second, and this was true even if the objects 
were moving at a relatively slow speed. This temporal 
crowding limit on tracking becomes more severe as 
the number of targets increases. 11) 


As the spatial, temporal, and speed limits are ap- 
proached, tracking performance decreases gradu- 
ally'7I"“l and in typical MOT displays, it is unclear 
which of these limits, or what combination of them, 
determine the maximum number of targets that can 
be tracked."°! For the spatial limit, one study found lit- 
tle to no effect beyond the Bouma's law crowding 
zone.!'”] Many MOT studies do not enforce sufficient 
spacing between objects to avoid spatial crowding, 
making spatial crowding likely to be one factor in over- 
all performance. 


Role of prediction and trajectory 
information 


Brains continuously predict some aspects of the fu- 
ture.°1°! Human object tracking performance is 
higher when object trajectories are predictable, sug- 
gesting that prediction also occurs in this domain. 
However, the benefit seems to disappear when there 
are more than one or two targets, '7°!'"4) suggesting that 
any prediction happening is more limited in processing 
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capacity than other aspects of object tracking. In those 
studies, however, predictability of objects' future posi- 
tions appears to be confounded with the objects being 
distinguishable from each other on the basis of main- 
taining particular and different motion directions. In 
such experiments, the difference in targets' and dis- 
tractors' motion directions or accelerations may be the 
facilitator of tracking rather than prediction of future 
positions.'’7! Indeed, distinctiveness of motion direc- 
tions alone facilitates tracking, '’”! thus any role played 
by the prediction of future positions remains unclear. 
However, performance detecting a change in a target's 
trajectory is much worse with each increase in target 
number, which suggests little ability to utilize trajec- 
tory information.'7*! 


Role of grouping and coordinate 
frames 


The human brain represents the positions of objects 
with multiple reference frames or coordinate systems. 
Early stages of the visual system represent the loca- 
tions of objects relative to the direction the eyes are 
pointing (retinotopic coordinates). Some later stages 
of human visual processing can represent object loca- 
tions relative to each other or to the scene. 


Regarding representation of relative locations, the rel- 
ative positions of objects can be represented with an 
imaginary polygon, with each target a different vertex 
of that polygon. In studies of MOT, Steve Yantis drew 
participants’ attention to the polygon formed by the 
targets and found that benefited performance, ”"! as 
did setting the targets’ trajectories to avoid much dis- 
ruption of the constantly-morphing polygon. This sug- 
gests that shape tracking contributes to accurate per- 
formance, at least in some participants.°! One study 
measured an electrical brain response (ERP) to a probe 
that was flashed while the objects were moving. The 
earliest-detectable part of the neural response to the 
probe was significantly greater if the probe lay on the 
polygon defined by the targets rather than inside or 
outside the polygon.'”°! This suggests that at least 
some of the participants continuously tracked the pol- 
ygon defined by the targets. 


Displays with more complicated statistical relation- 
ships among moving targets have been devised to 
show that regularities in hierarchical relationships are 
extracted and utilized in multiple object tracking, in- 
cluding nesting of groups of objects within moving ref- 
erence frames’?! 
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Updating of features other than 
position 


The classic MOT task requires updating of targets' po- 
sitions but not their other features. People appear to 
be less able to update the other features of targets, 
and have difficulty even in maintaining their 
knowledge of such features as the associated objects 
move. In one study, Pylyshyn assigned distinct identi- 
ties to four identical targets, either by giving them 
names or by giving them easily-identifiable starting 
positions: the four corners of the screen. In addition to 
the usual task at the end of the trial of identifying 
which objects were the targets, participants also were 
asked about the identity of the targets - which one 
each was. Contrary to what Pylyshyn expected from 
his FINST theory, accuracy at identifying which target 
was which was very low, even when accuracy reporting 
the targets’ positions was high.|?°! 


To assess maintenance of knowledge of object identi- 
ties, one series of experiments used cartoon animals as 
targets and distractors that all moved about the 
screen. By the end of each trial, the animals came to 
rest behind cartoons of cacti, so that their identities 
were no longer visible. Participants were asked where 
a particular target (e.g., the cartoon rabbit) had gone - 
that is, which occluder it was hiding behind. In this 
multiple identity tracking (MIT) task, performance was 
much worse than in the standard MOT task of report- 
ing target locations irrespective of which target a loca- 
tion belonged to.'*”! 


The deficit in updating the locations of featural and 
identity information may reflect a more general deficit 
in updating the locations of objects in visual short- 
term memory. In a study using a shell game in which 
the shells hid brightly-colored balls of wool, pairs of 
shells were swapped at a slow rate of once a second, 
but accuracy judging which shell contained a particular 
color fell to 80% accuracy when there were four swaps 
in a simple three-shell display, compared to 95% accu- 
racy for four swaps with a two-shell display.°°! 


The concept of an "object file" is that of a record in the 
brain that stores the features of a visual object, with 
the location record updated as the object moves." In 
the original studies that were motivated by this idea, 
one feature an object disappears and the object moves 
to a new location. The feature is then presented in the 
new location, and people respond faster to that fea- 
ture than to features that were not previously pre- 
sented as part of the object. This finding of priming in- 
dicates that an object file was created and updated by 
the brain. One might expect this to tap into the same 
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processing as that assessed by the MIT task. The rela- 
tionship between the two is unclear, however, as there 
is evidence that attentional tracking occurs can occur 
along a different trajectory than that which is the basis 
of updating the memory of an object's features. 7! 


In the studies mentioned so far, the objects involved 
did not change any of their features besides their posi- 
tions, so the task was to maintain knowledge of (un- 
changing) features while updating their positions. 
Change blindness studies show that in many circum- 
stances, people do poorly at noticing that features 
have changed. A famous demonstration involves plac- 
ing a blank screen between the presentation of two 
versions of a screen to mask the flicker that would oth- 
erwise be associated with a change. Change blindness 
also occurs when the flicker evoked by the change is 
masked by the objects' motion.'?!*"! That, however, 
may only mean that nothing is comparing the features 
present before and after the change; it does not neces- 
sarily mean that object representations are not up- 
dated, so other studies are needed. 


A related issue is whether tracking can occur on the 
basis not only of smooth changes in the position of an 
object, but also on the basis of smooth changes in an 
object's other features. In a tracking experiment in 
which two objects were always spatially superposed, 
the objects maintained their separate identities based 
on smooth continuity of their colors, orientations, and 
spatial frequencies. The participants could only track 
one such object! suggesting no ability to capitalize 
on spatiotemporal feature continuity for features other 
than position, although this has not yet been tested for 
cases in which the targets do not overlap (overlap may 
trigger figure-ground interference). 


Difficulty tracking unusual objects 
and object parts 


Many objects have clearly-visible parts. A dumbbell, 
for example, has a central bar part and has the weights 
at the bar's ends. Even when such parts are conspicu- 
ous, people can have difficulty tracking an individual 
part of multiple objects. When individual ends of multi- 
ple dumbbell-shaped drawings are designated as tar- 
gets, tracking performance is poor.°!"’! Performance 
was even worse when participants attempted to track 
one end of multiple moving lines, where the lines were 
uniform without distinct parts. Evidently, the mental 
processes that underlie tracking of multiple objects op- 
erate ona particular type of object representation that 
differs from what we can consciously recognize. Possi- 
bly the representation used for tracking is shared by 


40f 9 | WikiJournal of Science 


WikiJournal of Science, 2023, 6(1):3 
doi: 10.15347/wjs/2023.003 
Encyclopedic Review Article 


that used when searching for a particular colored 
shape that is hidden among many other shapes; visual 
search is hindered by connecting targets to distrac- 
tors. P2169 


For some types of "objects" that are not segmented as 
such by early visual processing, not even a single in- 
stance can be tracked. Stuart Anstis has shown that 
people are unable to track the intersection of two lines 
sliding over each other, except possibly at very slow 
speeds.!"”! 


Some things change shape as they move, such as liq- 
vids and slinkys. For slinky-like objects that moved by 
extending their leading edges to a point and then re- 
tracting their trailing edges, Kristy vanMarle and Brian 
Scholl found that tracking performance was 

poor." The underlying reason for this is unclear, but 
reporting the location of even a lone object is impaired 
by growth or contraction of the object, which may con- 
tribute to the tracking failure.°°! 


Interference with concurrent 
performance of other tasks 


Overlap among the processes underlying mental abili- 
ties can be revealed by what types of concurrent tasks 
interfere with each other. Attempting to track multiple 
visual objects typically interferes with other 

tasks, “7! even for tasks with stimuli in other modali- 
ties.“1] Unfortunately, it can be difficult to deter- 
mine whether this reflects processing somewhat spe- 
cific to our ability to track or instead reflects the pro- 
cessing necessary to initiate and sustain a wide variety 
of tasks. 


One exception to the usual finding of interference with 
other tasks is that an auditory pitch discrimination task 
was found to not interfere with visual multiple object 
tracking."“*! With a task designed as an auditory analog 
of tracking rather than just requiring discrimination of 
a few pitches, however, Daryl Fougnie et al. found that 
the task interfered approximately as much with visual 
object tracking as did a visual feature-tracking task. 
This suggests that auditory and visual tracking are lim- 
ited by acommon processing resource.'“°! 


Neural basis 


Neuroimaging studies find that activation of areas of 
the parietal cortex increases with the number of ob- 

jects tracked, which is consistent with the suggestion 
that the parietal cortex plays a role in humans' limited 
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tracking capacity.!7!""°!°! Activation of other brain ar- 


eas also seems to increase with target load, but the 
particular areas may be less consistent across studies 
than the parietal cortex finding. The size of partici- 
pants' pupils also increases with the number of objects 
tracked. The pupil size increase, which also is caused 
by mental effort in other tasks, may reflect norepi- 
nephrine release from the locus coeruleus.!“7!°°! 


Objects presented to the left visual hemifield are pro- 
cessed initially by the right cerebral hemisphere, while 
stimuli presented to the right visual hemifield are pro- 
cessed initially by the left cerebral hemisphere. The in- 
dependent capacity limits in the two hemifields are 
very similar, although there may be a small right-hemi- 
field advantage." A right hemifield advantage would 
be consistent with a contribution by both parietal cor- 
tices to tracking that hemifield, which was suggested 
because both parietal cortices are thought to contrib- 
ute to other attentional functions in the right hemi- 
field.64) 


The neural basis of MOT has also been studied us- 

ing electroencephalography (EEG). One such study 
found a robust correlation between tracking perfor- 
mance and the effect of number of targets on 

the N2pc event-related potential and also on contrala- 
teral delay activity.) Multiple brain areas contribute 
to these signals, so such studies have not yet allowed 
researchers to determine exactly which brain areas 
mediate tracking. 


Human variation and development 


The scores of a person are usually similar to each other 
if they are tested multiple times in the same MOT ex- 
periment (high task reliability). “9°!5°167 This suggests 
that the variation in the number of objects people 
seem able to track (for one version of the task, capaci- 
ties ranged between one and six targets)'""!"”! reflects 
real variation in ability. One caveat is that studies have 
failed to assess how much of this could be due to varia- 
tion in individuals' motivation. At least one study, how- 
ever, tested only top military recruits, a sample that 
was likely to be well-motivated.°*! 


Rather than samples from a broad population, most 
research has been conducted on undergraduates at 
universities in Western countries. Some studies, how- 
ever, have tested other populations. Comparing chil- 
dren of different ages, two studies in North America 
found a marked increase with age in the number of ob- 
jects the children could track, from 6 or 7 years old to 
adulthood.'°!'*!! For people with autism spectrum dis- 
orders, two studies found that such participants had 
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poorer MOT performance than typically-developing 
participants, which was attributed to a deficit in atten- 
tional selection.!°7!'! 


Adults with Williams Syndrome have profound deficits 
on certain spatial assembly tasks, such as copying a 
four-block checkerboard pattern.) For multiple ob- 
ject tracking, their performance is similar to typically- 
developing four- or five-year-old children.'°°!'°7!'! In 
contrast, their ability to remember the locations of 
MOT targets if they don't move is more comparable to 
typically-developing 6-year-olds, which has led to the 
suggestion that maintaining attentional selection is a 
particular problem in Williams Syndrome.'°°! 


Among older typically-developing adults, MOT perfor- 
mance falls steeply with age.!7°!'°’!!°°! Age-related in- 
creases in spatial crowding'®*! and temporal crowd- 
ing"*! likely contribute to this. 


Several papers report that video game players perform 
substantially better in MOT tasks than those who don't 
play video games.'°"!!”"! However, it has been sug- 
gested that this could be an artifact of research prac- 
tices such as selective publication of results!” 


Covariation of object tracking ability with 
other abilities 


While some have used MOT in an attempt to ensure 
study participants sustain their attention over along 
interval, a study with a large number of participants 
found little correlation with a continuous performance 
task specifically designed to measure lapses in atten- 
tion.'”7) MOT may, then, be forgiving of lapses in atten- 
tion, which is consistent with findings that for typical 
displays, participants can perform well in MOT even if 
they are occasionally briefly interrupted, with their 
tracking processes able to pick up where they left 

off, {421173 


One approach to investigating which tasks share un- 
derlying processing is to test participants on several 
different tasks to determine which tasks have the 
highest correlations across individuals. The results of 
studies that have done this with MOT have not been 
entirely consistent with each other, so which tasks 
yield the highest correlation with MOT performance is 
not yet clear. However, multiple studies find that visual 
working memory is one of the most highly-correlated 
tasks.“!'’“] The correlation is consistent with broader 
findings that working memory tasks are among the 
best predictors of performance in a wide range of 
tasks.|”°! Possibly this reflects shared underlying mech- 
anisms such as maintaining goal-relevant information 
in memory (possibly including which objects are the 
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targets) and disengaging from outdated or irrelevant 
information.” 


Use in ability testing and training 


Some professional sports teams use laboratory-style 
MOT tests for ability assessment and for train- 

ing.'””! Associates of the company that makes the 
"NeuroTracker" MOT product claim that NeuroTracker 
is a "cognitive enhancer" that improves a variety of 
abilities relevant to performance on the sports field, 
but the evidence in the studies purporting to show this 
is weak.'”*! Another reason for skepticism of such 
claims is the poor track record of other commercial 
"brain training" products advertised for their cognitive- 
enhancing effects. !”71l”°! 


While it is unlikely that training on laboratory-style 
MOT tasks yields broad mental benefits, when more 
rigorous studies are done, it is possible that firm evi- 
dence may support the use of tasks related to MOT for 
screening or training purposes for specific purposes. 
Regarding screening, however, one study found that 
laboratory MOT performance did not predict driving 
test performance as well as the Montreal Cognitive As- 
sessment, a trail-making task, or a useful field-of-view 
task.'°°! A multiple object avoidance (MOA) task, in- 
volving steering a ball with a computer mouse to pre- 
vent it from colliding with other moving balls ona 
computer screen, was found to correlate better with 
driving performance than MOT.'*" In another study, 
strong positive correlations with MOA performance 
were found with driving simulator performance and 
years of driving experience.'*?! This may be because 
MOA includes control of movement, which is neces- 
sary for driving, but is not required for MOT."*! 


Theories and models 


Published computational models fit some aspects of 
tracking results, with most focusing on the pattern of 
performance decline with increasing number of tar- 
gets, and some modeling the dissociation between po- 
sition and non-position features. No published theory 
purports to explain all four of the following: the diffi- 
culty with tracking parts of objects, the role of tem- 
poral interference, the dissociation between position 
and non-positional features, and the pattern of perfor- 
mance decline with increasing number of targets. 


Serial versus parallel processing 


The independence of tracking in the left and right 
hemifields suggests that position updating in each 
hemifield occurs independently of and in parallel with 
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position updating in the other hemifield (see the "Ca- 
pacity limits" section above). Within a hemifield, it is 
not yet completely clear whether tracking of multiple 
objects happens in parallel or instead the target posi- 
tions are updated one-by-one, but most recent theo- 
rists agree with Pylyshyn's original FINST theory that 
positions are updated in parallel.!°"!!°°!8°ll®71 4 finding 
that gives some support to the alternative of serial 
switching is the marked increase in temporal interfer- 
ence as the number of targets tracked increases. In 
particular, the amount of increase in time needed be- 
tween when a target leaves a location and a distractor 
takes its place is approximately predicted by the the- 
ory that attention must visit each moving target one- 
by-one to update its location. "“! 


Some who theorize that position updating occurs sim- 
ultaneously for multiple targets draw a contrast with 
features other than position, stating that they are up- 
dated by a process that must serially switch among the 
targets. &*1l®16ell871 A model of Lovett, Bridewell, & 
Bello published in 2019, for example, includes a paral- 
lel process to track changes in position and connect to 
visual pointers that are shared with visual short-term 
memory and other visual attention tasks. A serial se- 
lection process is also included, which operates on only 
one object at a time and enables access to a target's 
motion history and other features. °°! 


Slots versus resources 


Central to Pylyshyn's FINST theory is that a small set of 
discrete pointers mediate multiple object tracking. 
Subsequent researchers have suggested that rather 
than discrete pointers, a mental resource that is more 
continuous is divided among the targets. °°"! This dis- 
pute is similar to the "slots versus resources" debate in 
the study of working memory. A continuous resource 
naturally explains the smooth decline in performance 
with number of targets, although there is no agree- 
ment about what precisely about tracking becomes 
worse when less resource is provided. Possibilities in- 
clude spatial resolution, temporal resolution, the maxi- 
mum speed of the tracker, or all three (see the "Spatio- 
temporal limits" section above). 


Conclusion 


The apparent dissociation between knowledge of ob- 
ject locations and knowledge of their other features 
may be related to the role of attentional selection in 
feature binding. According to feature integration the- 
ory, serial attention is required to bind together indi- 
vidual non-position features with their locations. Be- 
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cause multiple object tracking appears to involve at- 
tending to multiple features simultaneously, it follows 
from feature integration theory that feature updating 
cannot be done for multiple objects. What was not 
necessarily expected from feature integration theory is 
that features or identifies of multiple moving objects 
cannot even be maintained. Treisman's contribution to 
influential foundational work on object files may have 
led many researchers to expect that once objects are 
bound, they would be maintained. Perhaps bound ob- 
ject files should be conceived of as being "hard to 
move". 


Serial attention may also be needed to create the full 
representations of objects that we are familiar with 
from introspection during focused attention. Without 
time for serial attention to visit individual objects, 
tracking may have to rely on proto-objects'*"! which 
could explain humans' difficulty in tracking parts of ob- 
jects: Poe) 


While tracking capacity is independent in the left and 
right visual hemifields, the neural basis of this has been 
hardly investigated, so we know very little about this 
hemisphere-specific visual attention process. For ex- 
ample, we do not know whether the interference doc- 
umented with other tasks is mediated by hemisphere- 
specific processes. The temporal crowding phenome- 
non described in the "Spatiotemporal limits" section is 
also relevant to interference with other tasks. Re- 
searchers rarely analyze how often target locations 
must be sampled to avoid confusion with distractors. 
For some of the MOT displays typically used, one 
might be able to attend to another task for hundreds 
of milliseconds and still be able to recover the targets 
based on proximity to their last location. Future stud- 
ies should use displays where the temporal crowding 
level is assessed and controlled, as in those of Hol- 
combe & Chen."““! This may result in higher correla- 
tions with tasks that measure whether attention is 
continuously sustained than in the study by Forten- 
baugh et al.!”7! 
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